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perational  test  and  evaluation 
“...does  not  include  an  operational 
assessment  based  exclusively  on —  (1) 
computer  modeling; 

(2)  simulation;  or  (3) 
an  analysis  of  . . .  program  documents.” 

(U.S.  Code  Title  10,  section  2399).  Yet 
we  use  not  only  realistic  tests  but  also 
models,  simulations,  and  the  discipline  of 
modeling  and  simulation  (M&S)  to 
inform  our  evaluations.  We  could  easily 
assume  a  lot  about  M&S,  but  intelligent 
use  of  M&S  in  T&E  requires  a  disci¬ 
plined  approach  in  both  planning  and 
confidence  building. 

Evaluators  often  rely  on  M&S  results 
to  garner  the  significance  of  test  results. 

Complexity  of  tests,  limited  test  articles 
and  test  range  assets,  and  safety  considerations  often 
drive  this.  In  a  systems-of-systems  test,  all  systems  or 
targets  need  to  be  present  or  represented.  Properly 
executed  tests  provide  the  data  upon  which  rest  the 
operational  assessment  of  the  system  and  the  validation 
of  its  models.  M&S  provides  a  measure  of  confidence 
that  sufficient  and  appropriate  tests  will  be  executed, 
supports  detailed  test  planning  and  rehearsals,  and 
facilitates  post- test  analyses. 

The  Department  of  Defense  (DoD)  Instruction 
5000.61  on  DoD  M&S  Verification,  Validation,  and 
Accreditation  (W&A)  was  signed  in  December  2009 
(and  is  considered  in  depth  in  another  article  in  this 
journal).  It  states  as  policy  that  “Models,  simulations, 
and  associated  data  used  to  support  DoD  processes, 
products,  and  decisions  shall...”  (a)  “...  undergo 
verification  and  validation  (V&V)  throughout  their 
lifecycles”  and  (b)  “...  be  accredited  for  an  intended 
use.”  Note  that  unlike  the  prior  version  of  the 
Instruction,  it  is  not  just  some  DoD  processes, 
products,  and  decisions  that  require  W&A  for  their 
intended  M&S  use. 

Proper  application  of  the  V&V  processes,  to  clearly 
show  model  assumptions,  limitations,  and  constraints, 
provides  confidence  that  we  can  accredit  their  use  for 
specific  purposes  in  T&E.  The  level  of  V&V  must  be 
appropriate  to  the  proposed  use,  and  to  the  maturity  of 
the  system  and  the  models.  In  general  terms,  less  V&V 


is  required  for  accreditation  of  uses  that  are  intended  to 
give  insight  into  trends  than  for  applications  intended 
to  support  decisions  upon  which  the  lives  of  those  in 
the  field  will  depend.  Some  intended 
uses  that  might  require  less  W&A  are 
for  data  to  indicate  testability  of  require¬ 
ments,  or  adequacy  of  the  infrastructure, 
or  early  test  planning.  Some  that  require 
more  V&V  include  use  of  M&S  to 
supplement  live  data  and  post-test  anal¬ 
ysis  of  results. 

Complex  systems  require  rigorous 
W&A  planning.  The  Missile  Defense 
Agency  recently  (spring  2010)  updated 
the  Integrated  Master  Test  Plan 
(IMTP)  for  the  Ballistic  Missile  Defense 
System  (BMDS).  The  IMTP  is  a  5-year 
plan  with  a  6-month  update  cycle.  This 
allows  it  to  identify  test  events  across  the  Future  Years 
Defense  Plan,  yet  be  responsive  to  results  of  test  events 
or  changes  in  executive  guidance. 

The  IMTP  is  data  driven  to  identify  the  critical  test 
factors — the  sensitive  data  elements  and  specific 
“critical  engagement  conditions” — needed  to  provide 
confidence  that  the  BMDS  and  its  system-level 
elements  are  accurately  represented.  It  also  identifies 
“empirical  measurement  events” — those  events  needed 
to  provide  confidence  in  system  performance  but  not 
necessarily  critical  to  M&S  verification  and  validation. 
To  date,  over  100  conditions  or  events  have  been 
identified  for  the  nine  system-level  elements  that  make 
up  the  BMDS. 

The  Honorable  Dr.  J.  Michael  Gilmore,  Director  of 
Operational  Test  and  Evaluation  (DOT&E),  approved 
this  IMTP.  He  stated  in  testimony  before  the  House 
Armed  Services  Committee,  on  April  15,  2010,  “If  the 
MDA  can  execute  the  revised  IMTP,  the  data  needed 
to  validate  models  and  perform  quantitative  assess¬ 
ments  of  BMDS  performance  will  become  available.” 

Naval  ships  and  submarines  are  among  the  most 
expensive  and  largest  procurements  in  the  DoD.  The 
first  unit  produced  is  an  operational  unit  rather  than  a 
prototype.  Yet  we  must  subject  them  to  “realistic 
survivability  testing” — “testing  for  vulnerability  of  the 
system  in  combat  by  firing  munitions  likely  to  be 
encountered  in  combat  (or  munitions  with  a  capability 
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similar  to  such  munitions)  at  the  system  configured  for 
combat...”  (U.S.  Title  10,  section  2366).  Waivers  for 
firing  munitions  at  the  first  units  have  been  granted  in 
all  cases  to  date,  but  alternative  test  plans  must  be 
approved  by  DOT&E. 

Since  1987,  live  fire  T&E  of  shock-hardened  (e.g., 
combat)  vessels  has  required  Full  Ship  Shock  Trials 
(FSST).  These  tests  help  evaluators  and  the  programs 
learn  lessons  that  can  improve  the  survivability  of  the 
vessels  subjected  to  the  shock  of  underwater  explosions. 
Lessons  learned  include  design  flaws,  fabrication 
deficiencies,  and  capabilities  of  systems  and  crews  to 
recover  mission  capability  after  such  an  event. 

A  FSST  detonates  several  charges  at  standoff 
distances  and  depths  to  create  shock  loadings  at 
fractions  of  the  design  criteria.  The  FSST  vessel  is 
fully  manned  and  underway  with  all  critical  systems 
operational.  The  vessel  is  first  “groomed” — for  exam¬ 
ple,  some  noncritical  less-rugged  systems  are  removed 
as  part  of  careful  preparation  and  inspections. 

State-of-the-art  simulations  (using  finite  element 
models)  offer  the  potential  to  augment  FSST  vulner¬ 
ability  and  survivability  data  with  data  that  cannot  be 
safely  obtained  from  a  FSST.  However,  real  questions 
remain  regarding  the  credible  use  of  M&S  to 
determine  when  equipment  failure  occurs  as  a  result 
of  shock  loading. 

The  Virginia  submarine  class  FSST  cancellation  was 
approved  by  DOT&E.  The  decision  was  based  on  its 
highly  shock-hardened  design,  W&A  of  the  trans¬ 
ient  shock  analysis  method,  and  tests  at  the  Aberdeen 
Test  Center  Underwater  Explosive  Test  Facility. 
These  tests  included  a  full-scale  model  of  approxi¬ 
mately  one  third  of  the  weapons  handling  module  at 
shock  levels  higher  than  those  normally  achieved  in 
FSSTs.  This  testing  revealed  things  that  a  FSST  could 
not  have  done. 

The  CVN  78  program  is  developing  an  advanced 
modeling  and  simulation  capability  to  augment 
reduced  scope  shock  testing  of  CVN  78.  The  Navy 
has  developed  a  memorandum  of  agreement  on  the 
elements  of  the  process.  DOT&E  intends  to  withhold 
its  decision  to  concur  with  the  implementation  of  the 
memorandum  of  agreement  until  the  feasibility  of  the 
M&S  is  demonstrated.  This  is  anticipated  in  FY13. 

An  alternative  approach  to  FSSTs  under  investiga¬ 
tion  is  use  of  compressed  air  guns  to  replace  the 


explosives  as  the  elements  that  mechanically  excite  the 
vessel.  Unlike  explosives,  compressed  air  guns  can  be 
reused.  However,  there  are  differences.  Shock  waves 
from  air  gun  arrays  are  currently  less  abrupt  and 
energetic  than  the  explosives.  The  mechanical  respons¬ 
es  are  initially  more  localized  and  then  propagate 
through  the  length  of  the  vessel.  It  may  be  appropriate 
to  conduct  some  side-by-side  full  ship  testing  using 
explosives  and  the  air  guns,  with  parallel  analyses  to 
investigate  differences  in  the  resulting  data.  This,  of 
course,  would  need  to  be  followed  with  analyses  to 
assess  the  significance  to  live  fire  testing  and  evaluation 
and  to  inform  future  decisions  on  intelligent  use  of  the 
simulated  explosive  test. 

Limitations  on  the  use  of  full  ship  shock  trials 
coupled  with  the  lessons  learned  from  the  mining  of 
the  USS  Princeton  during  the  First  Gulf  War  have 
spurred  the  Navy  to  develop  detailed  models  of  shock 
propagation  in  the  ship’s  structure  and  for  the 
prediction  of  damage  propagation  as  a  consequence 
of  weapon  effects.  These  support  the  assessment  of 
ship  survivability  for  live  fire  testing  and  evaluation. 

T&E  practices  involve  the  intelligent  use  of  M&S 
in  the  planning,  conduct,  and  analyses  of  test  events 
as  well  as  to  augment  these  test  data.  Credible  use  of 
M&S  in  support  of  T&E  requires  a  disciplined  ap¬ 
proach  to  establish  the  level  of  V&V  needed  for 
confident  accreditations  for  the  variety  of  intended 
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